02. What is Optimization?

L4 02 What Is Optimization V2

Second-order condition

In the previous problem, we cheated a little. We knew the shape of the function, and we knew its orientation from our plot, so when we found the point where the derivative equaled 0, we knew we had found the minimum. However, in general, points where the derivative equals 0 could be minima, maxima, or saddle points . To distinguish between these cases, we need to check the function's curvature around the point in question. We do this using the second derivative of the function. For a function of one variable, the rule is:

  • If \frac{\mathrm{d}^2y}{\mathrm{d}x^2}(x_0) < 0 then f has a local maximum at x_0 .
  • If \frac{\mathrm{d}^2y}{\mathrm{d}x^2}(x_0) > 0 then f has a local minimum at x_0 .
  • If \frac{\mathrm{d}^2y}{\mathrm{d}x^2}(x_0) = 0 , the test is inconclusive.

So in the case of the function above, we have

\frac{\mathrm{d}y}{\mathrm{d}x} = 2(x-1)

so,

\frac{\mathrm{d}^2y}{\mathrm{d}x^2} = 2

The second derivative is positive for all x , which means the function's slope is increasing everywhere, hence the function's curvature is upward everywhere. So we can be confident that the point we found is a minimum.

For a function of two variables, the rule must change a bit. We construct the matrix of second-order partial derivatives:

H(x,y)={\begin{pmatrix}f_{xx}(x,y)&f_{xy}(x,y)\\f_{yx}(x,y)&f_{yy}(x,y)\end{pmatrix}}

This is called the Hessian matrix H(x,y) .

Recall that the determinant of a matrix,

\mathrm{det}{\begin{pmatrix}a&b\c&d\end{pmatrix}}=ad - bc.

If the first-order partial derivatives are 0 at the point (a,b) , i.e. f_x(a,b) = f_y(a,b) = 0 , then, we apply the following rule:

  • If \mathrm{det}(H)(a,b)>0 and f_{xx}(a,b)>0 then (a,b) is a local minimum of f .
  • If \mathrm{det}(H)(a,b)>0 and f_{xx}(a,b)<0 then (a,b) is a local maximum of f .
  • If \mathrm{det}(H)(a,b)<0 then (a,b) is a saddle point of f .
  • If \mathrm{det}(H)(a,b)=0 then the second derivative test is inconclusive, and the point (a, b) could be any of a minimum, maximum or saddle point.

Next Concept